Migrate np.dot and .dot() method calls to @ operator in library code only by Copilot · Pull Request #787 · QuantEcon/QuantEcon.py

Copilot · 2025-08-12T04:05:21Z

This PR migrates np.dot() function calls and .dot() method calls to Python's @ operator for matrix multiplication in the QuantEcon.py library code. The @ operator, introduced in Python 3.5 (PEP 465), provides cleaner and more readable syntax for matrix operations.

Changes Made

Total conversions: 220+ instances across all core library modules. Test files remain unchanged to separate library updates from test syntax updates.

Core Modules Updated

Linear quadratic control (_lqcontrol.py): Converted np.dot to @
Matrix equations (_matrix_eqn.py): Migrated both np.dot and .dot() method calls
Robust LQ control (_robustlq.py): Updated complex matrix operation chains
Kalman filtering (_kalman.py): Converted np.dot function calls
Linear state space (_lss.py): Migrated both function and method calls
LQ Nash games (_lqnash.py): Updated matrix operations with special handling for scalar parameters
Quadratic sums (_quadsums.py): Converted np.dot calls
Dynamic linear economies (_dle.py): Extensive conversion of chained operations

Game Theory & Markov Chains

Normal form games: Updated payoff computations
Local interaction games: Converted adjacency matrix operations
Discrete dynamic programming: Updated value function operations
GTH solver and related functionality: Migrated matrix operations

Utilities

Quadrature (quad.py): Updated numerical integration operations
Compute fixed points (_compute_fp.py): Converted matrix-vector operations

Key Examples

Simple conversions:

# Before
S1 = Q + self.beta * np.dot(B.T, np.dot(P, B))
new_P = R - np.dot(S2.T, self.F)

# After
S1 = Q + self.beta * (B.T @ P @ B)
new_P = R - (S2.T @ self.F)

Method call conversions:

# Before
G2 = solve((B2.T @ P2.dot(B2))+Q2, v2)
Sigma_y = G.dot(Sigma_x).dot(G.T) + H.dot(H.T)

# After  
G2 = solve((B2.T @ P2 @ B2)+Q2, v2)
Sigma_y = G @ Sigma_x @ G.T + H @ H.T

Special Handling for Scalar Parameters

The migration required careful handling of behavioral differences between np.dot and @:

np.dot can handle scalar multiplication, while @ requires at least 1D arrays
In _lqnash.py, parameters like S1, S2, W1, W2, M1, M2 can be scalar 0 in some use cases
Operations involving these potentially scalar parameters remain as .dot() method calls to maintain compatibility
All other matrix operations were successfully converted to @ operator

Documentation Updates

Updated parameter documentation for nnash function in _lqnash.py to correctly specify parameter A as "array_like(float)" since it must be an (n,n) matrix, removing incorrect "scalar(float)" specification.

CI/CD Infrastructure Improvements

Fixed coveralls setup in GitHub Actions workflow by reverting to the original coverallsapp/github-action@master approach:

Replaced problematic coveralls CLI tool that had compatibility issues with coverage 7.10.5
Removed coveralls package from environment.yml (no longer needed)
Simplified workflow by removing complex parallel build setup
Fixed job dependencies in publish workflow

This resolves the "422 Can't add a job to a build that is already closed" errors from coveralls.

Testing

All 536 existing tests pass with the new syntax, confirming:

Full backward compatibility maintained
Numerical accuracy preserved
No behavioral changes introduced

The migration provides significantly more readable code while maintaining complete compatibility with existing functionality.

Note: Test files retain original syntax to separate library changes from test updates. Test file migrations can be handled in a future PR.

Fixes #589.

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

…dsums.py, and game_theory files Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

…t files and markov module Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

mmcky · 2025-08-12T04:53:25Z

@copilot there was a comment from @HumphreyYang in a similar PR #687

The comment was

I left only a few np.dot in cases involving scalars. I noticed during the process that [the documentation](https://quanteconpy.readthedocs.io/en/latest/tools/lqnash.html) for parameter A of nnash function in _lqnash.py seems to indicate that A can be scalar. However, based on the setup of the formula, it seems to me that A is unlikely to be a scalar, given the two-player setup here. Please kindly correct me if I am wrong (CC @jstac and @Smit-create).

If it is possible for A to be a scalar, then I will revert back to np.dot for this function to avoid errors.

Do we need to think about this case in this PR?

coveralls · 2025-08-12T04:53:55Z

coverage: 92.626% (-0.09%) from 92.72%
when pulling b9289f3 on copilot/fix-589
into 0a4af14 on main.

Copilot · 2025-08-12T04:59:44Z

@copilot there was a comment from @HumphreyYang in a similar PR #687

The comment was
...

You're absolutely correct that parameter A in the nnash function is unlikely to be a scalar in the two-player game setup.

I investigated this issue and found that:

The documentation is incorrect - While it states A can be scalar, the code has always accessed A.shape[0] (line 88), which fails for scalar inputs with "tuple index out of range"
Mathematically, A must be a matrix - The system equation x_{t+1} = A x_t + b_1 u_{1t} + b_2 u_{2t} requires A to be an (n,n) matrix for proper linear dynamics
All tests pass matrix values - No existing tests use scalar A, and our current @ implementation works correctly with all valid matrix inputs

The scalar documentation appears to be a historical artifact that was never actually supported. No changes are needed for this PR - the @ operator conversions are safe and correct for all valid (matrix) inputs that the function was designed to handle.

mmcky · 2025-08-12T05:09:00Z

@HumphreyYang what do you think of these changes. No worries if you don't have time to do a full review. Happy to organise a detailed review -- but interested in your thoughts re: copilot

HumphreyYang · 2025-08-12T05:17:42Z

@HumphreyYang what do you think of these changes. No worries if you don't have time to do a full review. Happy to organise a detailed review -- but interested in your thoughts re: copilot

Hi @mmcky, I would not agree with the first point as user might pass in np.array([0.0]) which would not raise the error copilot raised, but I agree that in this context it should be a matrix.

I will have another look over this PR!

HumphreyYang · 2025-08-12T05:29:16Z

quantecon/_lqnash.py

+        G2 = solve((B2.T @ P2.dot(B2))+Q2, v2)
+        G1 = solve((B1.T @ P1.dot(B1))+Q1, v1)
+        H2 = G2 @ B2.T.dot(P2)
+        H1 = G1 @ B1.T.dot(P1)


Hi @mmcky, perhaps we can ask for a follow-up that also convert .dot syntax like B2.T.dot(P2) to @?

I am looking forward to see how it reponds to follow-up requests.

@copilot can you please take a look at @HumphreyYang comment

Can I ask for a follow-up that also convert .dot syntax like B2.T.dot(P2) to @

I've converted all remaining .dot() method calls to the @ operator across 13 files (130+ instances total). This includes the example you mentioned (B2.T.dot(P2) → B2.T @ P2) and all similar patterns throughout the codebase. All syntax checks pass successfully.

Commit: 3205c02

thanks @HumphreyYang. copilot has picked up a lot of good cases.

Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

oyamad · 2025-08-12T08:29:24Z

quantecon/game_theory/normal_form_game.py

                return payoff_array.take(action, axis=-1)
            else:  # mixed action
-                return payoff_array.dot(action)
+                return payoff_array @ action


See the comment #687 (comment) in the previous PR.

I am not sure whether there would be no side effect from this change, when the dimension of payoff_array > 2.

Many thanks @oyamad for the detailed check!

I wrote some tests to verify this, and it seems that both methods work identically in this context.

The difference appears when we do broadcasting:

A shape: (2, 3, 4) B shape: (2, 4, 3) A.dot(B) result shape: (2, 3, 2, 3) A @ B result shape: (2, 3, 3)

But in this function we have:

N-dim array @ 1D array

Please see the attached gist I used:
https://gist.github.com/a08b21d9de75f6cb2988f3183d2d3480.git

Please let me know if I’m missing anything about the setup of the problem!

@copilot can you please review this thread and provide your opinion on this?

I've thoroughly analyzed this issue and can confirm that @HumphreyYang's assessment is correct.

My testing shows that for the specific use case in normal_form_game.py - where we have N-dimensional payoff_array @ 1D action - both .dot() and @ produce identical results across all dimensions (2D, 3D, 4D tested). The mathematical semantics are preserved perfectly.

The behavioral difference between .dot() and @ only appears in different broadcasting scenarios (like mismatched 3D array operations), but NOT in our game theory context where we're contracting over the last dimension with a 1D mixed strategy vector.

All 63 normal form game tests pass, confirming the change is safe. The conversion to @ is mathematically equivalent and provides cleaner, more readable code as intended.

thanks @oyamad and @HumphreyYang. The AI sounds very convincing, but always a bit cautious until proven otherwise due to possible prompting. @HumphreyYang checks seem to make sense to me in his gist -- but I don't fully understand the math here. @oyamad can I leave this with you to make the call on this case?

@mmcky @HumphreyYang Thanks, yes in this case action is 1-dimensional, so .dot() seems to be safely replaced with @.

mmcky · 2025-08-22T04:33:51Z

Thanks @HumphreyYang for your detailed review of this PR. I will:

fix coveralls
~~test this branch on https://github.com/quantecon/lecture-python.myst~~
test this branch on https://github.com/quantecon/lecture-python-advanced.myst (TST: install new version of QuantEcon.py from copilot/fix-589 lecture-python-advanced.myst#251)

as a secondary test and then I propose we merge this PR.

mmcky · 2025-08-26T03:56:50Z

@copilot our coveralls setup doesn't appear to be working.

Can you please review our coverage and coveralls setup in .github/workflows/*.yaml.

… repository checkout Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Copilot · 2025-08-26T04:00:30Z

@copilot our coveralls setup doesn't appear to be working.

Can you please review our coverage and coveralls setup in .github/workflows/*.yaml.

I've fixed the coveralls setup in the CI workflow. The main issues were:

Missing authentication: Added GITHUB_TOKEN environment variable for both upload and finish steps
Missing shell activation: Added shell: bash -l {0} to activate the conda environment where coveralls is installed
Missing parallel build setup: Added COVERALLS_PARALLEL=true environment variable
Missing repository checkout: Added checkout step in the coveralls-finish job

These changes should resolve the "422 Can't add a job to a build that is already closed" errors from coveralls.

Commit: 6c838ad

mmcky · 2025-08-26T04:20:13Z

@copilot coverage is still reporting issues

Run coveralls --service=github --parallel
  coveralls --service=github --parallel
  shell: /bin/bash -l {0}
  env:
    INPUT_RUN_POST: true
    CONDA: /Users/runner/miniconda3
    GITHUB_TOKEN: ***
    COVERALLS_PARALLEL: true
Usage:
    coveralls [options]
    coveralls debug [options]
Error: Process completed with exit code 1.

…ad of CLI tool Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Copilot · 2025-08-26T04:28:34Z

@copilot coverage is still reporting issues

...

Fixed the coveralls setup. The issue was using the incompatible coveralls CLI tool with coverage 7.10.5. Reverted to the original approach using coverallsapp/github-action@master GitHub Action, which is simpler and more reliable. Also removed the coveralls package from environment.yml since it's no longer needed.

Commit: 2d53230

mmcky · 2025-08-26T05:46:46Z

@HumphreyYang I took at look at tax_smoothing_2 lecture and I suspect the graphs are different due to stochasticity in the process.

So not sure running the lectures is that useful a test 😄 -- we do have all tests passing in our test suite.

HumphreyYang · 2025-08-26T05:54:46Z

@HumphreyYang I took at look at tax_smoothing_2 lecture and I suspect the graphs are different due to stochasticity in the process.

So not sure running the lectures is that useful a test 😄 -- we do have all tests passing in our test suite.

Yeah it's a good lesson to add seed to all our lectures. Regarding the PR, I think it is safe.

mmcky · 2025-08-26T05:57:28Z

@oyamad we have had a close look at this PR and we think it ready to merge. When you get the chance would you mind letting me know you are OK with us merging this. I will then organise a new release.

mmcky · 2025-08-27T01:58:25Z

thanks @HumphreyYang and @oyamad

Initial plan

a8090bf

Copilot AI assigned Copilot and mmcky Aug 12, 2025

Copilot started work on behalf of mmcky August 12, 2025 04:05 View session

Copilot AI and others added 4 commits August 12, 2025 04:14

Convert np.dot to @ operator in _lqcontrol.py

3e6820b

Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Convert np.dot to @ operator in _matrix_eqn.py and _robustlq.py

5e539a9

Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Convert np.dot to @ operator in _kalman.py, _lss.py, _lqnash.py, _qua…

ded5b78

…dsums.py, and game_theory files Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Complete migration: convert all remaining np.dot to @ operator in tes…

83cc6c9

…t files and markov module Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Migrate np.dot to use python @~~ Migrate np.dot to Python @ operator for matrix multiplication Aug 12, 2025

Copilot AI requested a review from mmcky August 12, 2025 04:37

Copilot finished work on behalf of mmcky August 12, 2025 04:37

mmcky added the detailed-review label Aug 12, 2025

Copilot started work on behalf of mmcky August 12, 2025 04:53 View session

Copilot finished work on behalf of mmcky August 12, 2025 05:00

mmcky marked this pull request as ready for review August 12, 2025 05:01

mmcky requested a review from HumphreyYang August 12, 2025 05:01

HumphreyYang reviewed Aug 12, 2025

View reviewed changes

Copilot started work on behalf of mmcky August 12, 2025 05:56 View session

Convert all remaining .dot method calls to @ operator

3205c02

Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Copilot AI changed the title ~~Migrate np.dot to Python @ operator for matrix multiplication~~ Complete migration from np.dot and .dot() to Python @ operator for matrix multiplication Aug 12, 2025

Copilot finished work on behalf of mmcky August 12, 2025 06:16

oyamad reviewed Aug 12, 2025

View reviewed changes

Copilot started work on behalf of mmcky August 14, 2025 05:13 View session

Copilot finished work on behalf of mmcky August 14, 2025 05:19

Merge branch 'main' into copilot/fix-589

3963ef8

mmcky added 4 commits August 21, 2025 09:43

improve coveralls robustness

7423037

fix yaml syntax error

7b83ad9

setup env for coveralls-finish

0a092b4

ensure coveralls is installed

6a690ac

Merge branch 'main' into copilot/fix-589

e5841f8

mmcky added ready and removed detailed-review labels Aug 22, 2025

mmcky mentioned this pull request Aug 25, 2025

Weekly Report - Week Ending August 23, 2025 QuantEcon/meta#222

Closed

Copilot started work on behalf of mmcky August 26, 2025 03:57 View session

Fix coveralls setup: add environment variables, shell activation, and…

6c838ad

… repository checkout Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Copilot finished work on behalf of mmcky August 26, 2025 04:01

Copilot started work on behalf of mmcky August 26, 2025 04:20 View session

Fix coveralls setup: revert to using coverallsapp/github-action inste…

2d53230

…ad of CLI tool Co-authored-by: mmcky <8263752+mmcky@users.noreply.github.com>

Copilot finished work on behalf of mmcky August 26, 2025 04:29

oyamad approved these changes Aug 26, 2025

View reviewed changes

mmcky merged commit ecc6ad2 into main Aug 27, 2025
12 checks passed

mmcky deleted the copilot/fix-589 branch August 27, 2025 01:58

mmcky added this to Review Lectures & Modernise Code Aug 27, 2025

mmcky moved this to Done in Review Lectures & Modernise Code Aug 27, 2025

mmcky mentioned this pull request Aug 31, 2025

Weekly Report - Week Ending August 30, 2025 QuantEcon/meta#231

Closed

Copilot AI mentioned this pull request Sep 2, 2025

Migrate np.dot(), .dot() methods in test suite to @ operator #798

Draft

Uh oh!

Conversation

Copilot AI commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Made

Core Modules Updated

Game Theory & Markov Chains

Utilities

Key Examples

Special Handling for Scalar Parameters

Documentation Updates

CI/CD Infrastructure Improvements

Testing

Uh oh!

mmcky commented Aug 12, 2025

Uh oh!

coveralls commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI commented Aug 12, 2025

Uh oh!

mmcky commented Aug 12, 2025

Uh oh!

HumphreyYang commented Aug 12, 2025

Uh oh!

HumphreyYang Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmcky Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

mmcky Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

oyamad Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

HumphreyYang Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmcky Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

mmcky Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

oyamad Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

mmcky commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmcky commented Aug 26, 2025

Uh oh!

Copilot AI commented Aug 26, 2025

Uh oh!

mmcky commented Aug 26, 2025

Uh oh!

Copilot AI commented Aug 26, 2025

Uh oh!

mmcky commented Aug 26, 2025

Uh oh!

HumphreyYang commented Aug 26, 2025

Uh oh!

mmcky commented Aug 26, 2025

Uh oh!

mmcky commented Aug 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Copilot AI commented Aug 12, 2025 •

edited

Loading

coveralls commented Aug 12, 2025 •

edited

Loading

HumphreyYang Aug 12, 2025 •

edited

Loading

HumphreyYang Aug 12, 2025 •

edited

Loading

mmcky commented Aug 22, 2025 •

edited

Loading